With the explosion of desktop publishing, the availability of digital media, and the advent of the World Wide Web (WWW), it is now possible to access large multimedia document repositories distributed throughout the world. Such access is becoming increasingly important in a number of applications including medical diagnostics, manufacturing, pharmaceutical research, surveillance, and distributed publishing of multimedia data from repositories. While data from widely distributed sources is becoming easily accessible, this also poses challenges to search agents at servers demanding them to do intelligent site selection based on the integration of information from database sites and multimedia information in a query. Resource discovery systems in information retrieval can mostly handle textual queries and text databases. The problem of site selection based on image content information in the query has not yet been addressed.
Accessing repositories in a distributed setting, either over a proprietary network or the more public Internet and its instantiation, the World Wide Web (WWW), poses quite a few challenges. In a typical scenario for such systems, the access to multimedia databases at remote web sites may be initiated by a client machine running a browser product such as Netscape Navigator.TM. (by Netscape Communications Corporation) or Internet Explorer.TM. (by MicroSoft Corporation). The query is processed by the browser and sent to a web server. The web server selects the target multimedia database site and poses the query to the database in an acceptable form. The indexing mechanism of the database searches its repository for possible answers to the posed query. The answer is fed back to the web server for eventual relaying to the client. As can be seen from this scenario, system issues of security, consistency, versioning, persistence, etc., that are relevant for distributed access to traditional databases are also relevant for multimedia databases. More importantly, new technical issues arise due to the nature of the multimedia data. The traditional way of accessing such databases by assigning text annotations to image and video data are insufficient to unambiguously describe the media content, not to mention the time-consuming task of manually tagging the multimedia data. Automatic database creation and image content-based indexing, on the other hand, are difficult problems, for which effective methods are yet to emerge. While it is true that these problems exist also for multimedia databases designed for a stand alone use, they are felt more keenly when such databases are designed for use in a distributed (client-server) setting. Traditional methods of designing such systems using proprietary methods of query specification and data organization and search for specific fixed queries may not be suitable in such a setting where the remote user's queries may be unanticipated and referring to image content, yet unextracted.
Even when individual databases are designed in a consistent manner, other issues of web-based access have to be taken into consideration. In particular, the process of selecting the relevant databases for a given query remains a challenging problem. Even though the number of multimedia databases linked by the network may be fewer than text databases currently (although this may change with the increased popularity of the WWW), it is still crucial to perform a careful selection of database sites for computational reasons, particularly because of the inherent complexity in image-content-based querying of each database. Secondly, if a query is posed to several databases, the answers may need consolidation and summarization before they can be presented to a user. Finally, the transformation of a user query (who may not be aware of the databases and their capabilities) into a form suitable for querying the remote multimedia databases may require eliciting more information from the user than that provided in the original query.
Although commercial systems are being developed that allow multiple text databases to be accessed over the web via languages such as SQL, ODBC or Perl gateways, as discussed by John K. Whetzel in "Integrating the World Wide Web and Database Technology", AT&T Technical Journal, pages 38-46, March/April 1996, no methods currently exist that allow a user to interact with multiple remotely located multimedia database systems for image content queries. The systems that come close are those that allow access to image collections (rather than databases) like the University of Chicago's WebSeer.TM. system as discussed by Michael Swain et al. in "WebSeer: An Image Search Engine for the World Wide Web", IEEE Workshop on Content-based Access of Image & Video Libraries, San Juan, June 1997, Columbia University's WebSEEK system as described in a paper by John Smith and Shih-Fu Chang, in "VisualSEEk content-based image/video database system", Advent Project--School of Engineering and Applied Science Activity Report, pages 3.1-3.2, Columbia University 1996, and Berkeley digital library project CYPRESS as described in a paper by V. Ogle and M. Stonebraker in "Chabot: Retrieval from a Relational Database of Images", IEEE Computer, pages 40-48, September 1995 and a paper by D.Forsyth et al. entitled "Searching for Digital Pictures", Scientific American, pages 88-92, June 1997. WebSeer is modeled after traditional search engine products such as Lycos.TM. (by CMU) and Alta Vista.TM. (by Digital Corporation) in that it also creates its own indexed database at the web server site by navigating known web sites and recording text as well as image related information. More common are systems like CYPRESS and QBIC, as described by M. Flickner et al., "Query by Image and Video Content: The QBIC System", IEEE Computer, 28 (9), pages 23-30, 1995, that allow web users to use their pre-designed image databases by connecting to their specific URL. Furthermore, such systems often reflect closed design strategies that are optimized for handling, a fixed set of queries using representations all pre-computed at the time of database creation. Being client-centric, they offer little capability for sharing and a potential for duplication and inconsistency.
Previous work does exist, however, on addressing individual aspects of web-based multimedia databases, particularly, in the design of image and video databases, and resource discovery systems. Issues of image data modeling and image matching have been explored in the prior art. These subjects are discussed in the following publications: R. Jain and S .N .J. Murthy, "Similarity Measures for Image Databases", Proceedings of the SPIE Conference on Storage and Retrieval of Image and Video Databases III, pages 58-67, 1995; and the Flickner et al. paper cited above. These issues were, however, examined in isolation in the context of the related applications with no coherent design framework emerging. In particular, the integration of such database systems over the Internet, and its instantiation, the World Wide Web, has not been explained in prior work.
Most of the site selection work has been focused towards handling text information. For example, web search engines such as Lycos.TM. and Alta Vista.TM. currently create web indices in their search engines by periodically scanning potential web sites and using the text information in their resident HTML pages. But most implementations of text-based distributed systems do not perform site selection, often posing a query to all sites in parallel as done in the CLASS (College Library Access and Storage System) and the NCSTRL (Networked Computer Science Technical Report Library) systems at Cornell University as discussed by C. Lagoze and J. Davis, "Dienst: An architecture for distributed document libraries", Communications of ACM, 38(4), page 47, April 1995. More recently, techniques from information retrieval are being used for intelligent resource site selection. For textual data, examples of such systems include GLOSS (from Stanford) as discussed by Luis Gravano and H. Garcia-Molina, "Generalizing Gloss to Vector-Space Databases and Broker Hierarchies," Proceedings of the 21st International Conference on Very Large Data Bases, pages 78-89, 1995, WHOIS++ (available from Bunyip's Internet Services Technologies Group) and HARVEST (from University of Colorado) as discussed Michael Schwartz, "Internet Resource Discovery at the University of Colorado", IEEE Computer, pages 25-35, September 1993. A further discussion of information resource discovery systems on the Internet is provided by Katia Obraczka et al. in "Internet Resource Discovery Services", IEEE Computer, pages 8-22, September 1993. These systems employ statistical approaches to record the frequency of occurrence of text keywords from known sites to construct an index of relevant sites for directing a query. In particular, the GLOSS (generalized glossary of servers) server keeps statistics on the available databases to estimate which databases are potentially most useful for a given query using Boolean and vector-space retrieval models of document retrieval. Another approach has been to use inference networks for the text database discovery problem as discussed in a paper by J. Callan et al., "Searching Distributed Collections with Inference Networks", Computer Science Dept., University of Massachusetts, 1995. This summarizes databases using document frequency information for each term together with the inverse collection frequency of different terms. An inference network then uses this information to rank the databases for a given query. Finally, the HARVEST system provides a flexible architecture for accessing information over the Internet using "gatherer" modules to collect information about the data resources which is passed to the "broker" modules. A structured representation of these broker modules is kept in the Harvest server registry which in a sense, becomes the meta-database exposing information about the individual database sites.
The present invention employs network capabilities to achieve various advantageous ends. The foregoing references are intended to provide a background for any appropriate network implementation required by the disclosed embodiment below for the purpose of rendering or transporting database information. Disclosures of all of the references cited and/or discussed above in this Background are incorporated herein by reference.